Adding LITTLEFS w/o rebuild the IDF #4096

lorol · 2020-06-17T20:31:07Z

based on https://github.com/joltwallet/esp_littlefs

This will be my first PR, hope will do it right.
Details in README

based on https://github.com/joltwallet/esp_littlefs This will be my first PR, hope will do it right.

Anyway, the LITTLEFS is a hack.

…e the PR status here espressif/arduino-esp32#4096

…e the PR status here: espressif/arduino-esp32#4096

atanisoft

since this is embedding https://github.com/joltwallet/esp_littlefs it would be good to have sign-off from @joltwallet / @BrianPugh since some files are showing mixed/missing licensing.

This PR will also need to be updated for the esp32s2 branch which is based on IDF v4.2 (master) as there are VFS API changes which may break this functionality.

atanisoft · 2020-06-18T12:03:18Z

package/package_esp32_index.template.json

-    }
-  ]
-}
+{


reformatting of an existing file (such as this one) is often discouraged and will often lead to the PR sitting for a very long time (if it ever gets merged).

If you do reformat the file, please ensure it is consistent as it doesn't appear to be consist after reformatting (looks like a mix of spaces and tabs)

@joltwallet and @BrianPugh
Could you please help in this?

atanisoft · 2020-06-18T12:07:04Z

libraries/LITTLEFS/src/LITTLEFS.cpp

+//esp_err_t esp_littlefs_format(const char* partition_label);
+//esp_err_t esp_littlefs_info(const char* partition_label, size_t *total_bytes, size_t *used_bytes);
+
+#define LFS_NAME "spiffs"


instead of a hardcode via DEFINE can you try:

static constexpr const char LFS_NAME[] = "spiffs";

This will prevent global namespace pollution.

atanisoft · 2020-06-18T12:09:08Z

CMakeLists.txt

+libraries/LITTLEFS/src/esp_littlefs.c
+libraries/LITTLEFS/src/lfs.c
+libraries/LITTLEFS/src/lfs_util.c
+libraries/LITTLEFS/src/littlefs_api.c


please correct formatting on these additions.

atanisoft · 2020-06-18T12:12:08Z

libraries/LITTLEFS/src/LITTLEFS.cpp

+
+using namespace fs;
+
+class LITTLEFSImpl : public VFSImpl


@me-no-dev would it be feasible to provide a default impl for VFSImpl that can be used by the various FS impls rather than each one copying/pasting the same but with a different name?

what is the same?

nvm, this class can be dropped entirely.

atanisoft · 2020-06-18T12:12:57Z

libraries/LITTLEFS/src/LITTLEFS.cpp

@@ -0,0 +1,130 @@
+// Copyright 2015-2016 Espressif Systems (Shanghai) PTE LTD


adjust copyright year to 2020 (for all files with similar header)

lorol · 2020-06-18T14:58:08Z

since this is embedding https://github.com/joltwallet/esp_littlefs it would be good to have sign-off from @joltwallet / @BrianPugh since some files are showing mixed/missing licensing.

Asking Brian to review...

This PR will also need to be updated for the esp32s2 branch which is based on IDF v4.2 (master) as there are VFS API changes which may break this functionality.

Someone else should help here please to adapt the code to match older IDF, maybe @BrianPugh is the best to help with such branch of his library.

BrianPugh · 2020-06-18T17:11:47Z

Hey guys,

Some questions/feedback:

Instead of copying source files from my repo, could we submodule them (or even just include arduino-related files in my repo, so long as they don't conflict with esp-idf compilation). This way upstream bug-fixes and features are easily maintained.

My action plans:

Update the LICENSE in my repo (could have swore i had MIT licensing on it). The LittleFS project is BSD3 licensed, which seems to play nicely so thats no big deal. Looping back to point 1 in questions/feedback, some action like that will need to be taken so that these upstream changes apply to this project.
As for VFS API changes, can @atanisoft link me either a file/commit/pr/doc that specifies the changes? I should be able to accomodate fairly easily (if it requires changes on my side).

edit 1: Added MIT license to my repo

atanisoft · 2020-06-18T17:20:41Z

As for VFS API changes, can @atanisoft link me either a file/commit/pr/doc that specifies the changes? I should be able to accomodate fairly easily (if it requires changes on my side).

IDF v4.0 changed a few things: https://github.com/espressif/esp-idf/releases/tag/v4.0, there are likely more changes beyond v4.0 that would need to be handled. You can detect the IDF version via a few compile time defines.

BrianPugh · 2020-06-18T17:31:35Z

As for VFS API changes, can @atanisoft link me either a file/commit/pr/doc that specifies the changes? I should be able to accomodate fairly easily (if it requires changes on my side).

IDF v4.0 changed a few things: https://github.com/espressif/esp-idf/releases/tag/v4.0, there are likely more changes beyond v4.0 that would need to be handled. You can detect the IDF version via a few compile time defines.

ah gotcha, I've been regularly using my repo off of esp-idf master for esp32 targets, so it should be good to go. I thought there may be even newer or esp32-s2 specific changes.

atanisoft · 2020-06-18T17:37:13Z

I've been regularly using my repo off of esp-idf master for esp32 targets, so it should be good to go.

Great! I don't know of any changes specific to the ESP32-S2 that will prevent this from working.

lorol · 2020-06-18T17:48:38Z

@BrianPugh Hi Brian,

It will be difficult for me to re-wrap the LITTLEFS.cpp and .h in such a way that your library is kept as it is and just fork the appropriate revision of it.
As I said earlier, i just dropped the files and tried until it works with minimal spread around.
I know it is an ugly hack - but I didn't find existing implementation.
@me-no-dev is busy with other priorities and LittleFS is even not in the main focus of ESP32 (at all).
Otherwise, someone may follow the entire procedure of re-component-ing Arduino under IDF then re-packing it and dropping all new *.a files plus includes where they should be.
Personally I prefer independent than embedded through *.a files way, especially if it works same way.
Maybe I'm getting too old to learn new things.

No problem if you rewrite it entirely around your wrap, especially if it can be brought back to Arduino w/o painful recompiling and IDF syncing.

LL

atanisoft · 2020-06-18T17:54:22Z

No problem if you rewrite it entirely around your wrap, especially if it can be brought back to Arduino w/o painful recompiling and IDF syncing.

The best way to avoid that would be have @BrianPugh submit a PR to ESP-IDF to have LittleFS included as a component there. Then it can be wrapped in arduino-esp32 in a clean manner (like other FS types)

BrianPugh · 2020-06-18T18:06:57Z

I'll make a PR, no harm in trying. I fear that there might be a lot of other requirements (writing up a lot of documentation, etc)

lorol · 2020-06-18T18:12:11Z

I wish you a patience and good luck with that :)
Will go silent for a while.

BrianPugh · 2020-06-18T18:17:35Z

Here's a link to my PR for littlefs in esp-idf:

espressif/esp-idf#5469

lbernstone · 2020-06-18T18:25:11Z

If you've used the package, you know documentation is not always most thorough...
Having it as a separate module until is really robust is better than trying to get this in ASAP. Register it with Arduino and it will get just as much exposure as having it baked in.

BrianPugh · 2020-06-18T18:27:51Z

For registering, is that targeted at me or @lorol ? I don't really use arduino (and am unfamiliar with its package managers, library structures, and all that). I'm just here to help on actions requiring changes in my repo.

lbernstone · 2020-06-18T18:30:17Z

Targeted at OP (@lorol), whom wants it in arduino-esp32.

atanisoft · 2020-06-19T11:39:15Z

libraries/LITTLEFS/src/LITTLEFS.cpp

+
+using namespace fs;
+
+class LITTLEFSImpl : public VFSImpl


nvm, this class can be dropped entirely.

atanisoft · 2020-06-19T11:41:11Z

libraries/LITTLEFS/src/LITTLEFS.cpp

+    return (f == true) && !f.isDirectory();
+}
+
+LITTLEFSFS::LITTLEFSFS() : FS(FSImplPtr(new LITTLEFSImpl()))


Change this to:

LITTLEFSFS::LITTLEFSFS() : FS(FSImplPtr(new VFSImpl()))

and remove the definition of class LITTLEFSImpl. The customization is not necessary for LITTLEFS as the default behavior from VFSImpl() will work.

lorol · 2020-06-19T20:21:28Z

Targeted at OP (@lorol), whom wants it in arduino-esp32.

Thanks but ,,,
Please let's wait first IDF team to determine the strategy there.
Meanwhile, whoever desperately needs to test sketches can use from here or drop in libraries.
~~Hope we will have somehow IDF version dependency resolved (buy #define or so)~~
BTW, If .utime_p = &vfs_littlefs_utime, is removed, then it builds and works with core release 1.0.4 (IDF v.3.2) too

#define CONFIG_LITTLEFS_LOOKAHEAD_SIZE 128

- In esp_littlefs.c, added a way to build for older IDF too (IDF 3.2 is used on release 1.0.4) #define CONFIG_LITTLEFS_FOR_IDF_3_2

Modified Update library to allow FatFS OTA Updated file: package_esp32_index.template.json Added a sub-folder with IDE plugins

Updated esp32fs.jar and README for it

lorol · 2020-09-02T15:24:54Z

If you've used the package, you know documentation is not always most thorough...
Having it as a separate module until is really robust is better than trying to get this in ASAP. Register it with Arduino and it will get just as much exposure as having it baked in.

Request done: arduino/Arduino#10719

me-no-dev · 2020-11-02T20:21:17Z

Here is how we will go for 2.0.0:

I will add to the build system joltwallet/esp_littlefs so that it will be available
@lorol you can then change this PR to not have the lfs files and be internal as opposed to submodule (that have proven problematic in the past)
we will have LittleFS in ESP Arduino 2.0.0

me-no-dev · 2020-11-02T21:23:00Z

@lorol esp_littlefs has been added to the esp32s2 branch. You can try it there.

lorol · 2020-11-02T21:33:51Z

Ok, great! I see the lib.a and headers. Will try. I'll let you or Brian know if I need help to make it correct.

lorol · 2020-11-03T04:53:46Z

@me-no-dev
I created a folder in libraries and added the 2 files for test at a local clone.
It looks it may work as you requested :) just annoying warnings like:

hardware\espressif\esp32/tools/sdk/esp32/include/newlib/platform_include/sys/termios.h:97: warning: "B110" redefined
#define B110 3

hardware\espressif\esp32\cores\esp32/binary.h:65: note: this is the location of the previous definition
#define B110 6

hardware\espressif\esp32/tools/sdk/esp32/include/driver/include/driver/adc_common.h:368:11: note: declared here
esp_err_t adc2_vref_to_gpio(gpio_num_t gpio) attribute((deprecated));

Actually it builds w/o adding lfs files as submodule. I guess already built in through esp_littlefs?
I will test a bit more.
Any help to clean these warnings if possible?
Also since the branches are different, I don't see how to reuse this PR - maybe new one s2 to s2 ?

me-no-dev · 2020-11-03T10:05:12Z

@lorol lfs is already in. you only need the two LITTLEFS source files :)

to fix the redefine warning, I need the full error trace, so we can see where they are both defined. ADC I'll fix later.

me-no-dev · 2020-11-03T10:06:25Z

@lorol I'll suggest you open a pull request to the esp32s2 branch so we can give it a go (will need to add the tools, etc as well)

lorol · 2020-11-03T15:32:03Z

@me-no-dev @BrianPugh
Below is the Bxxxx redefinning chain.
If I comment the termios.h inclusion at esp_littlefs.h, warnings disapear. I asked Brian if termios is really necessary.

In file included from --\hardware\espressif\esp32/tools/sdk/esp32/include/esp_littlefs/include/esp_littlefs.h:16,
from --\hardware\espressif\esp32\libraries\LITTLEFS\src\LITTLEFS.cpp:23:
--\hardware\espressif\esp32/tools/sdk/esp32/include/newlib/platform_include/sys/termios.h:97: warning: "B110" redefined
#define B110 3

In file included from --\hardware\espressif\esp32\cores\esp32/Arduino.h:40,
from --\hardware\espressif\esp32\libraries\FS\src/FS.h:25,
from --\hardware\espressif\esp32\libraries\FS\src/vfs_api.h:18,
from --\hardware\espressif\esp32\libraries\LITTLEFS\src\LITTLEFS.cpp:17:
--\hardware\espressif\esp32\cores\esp32/binary.h:65: note: this is the location of the previous definition
#define B110 6

In file included from --\hardware\espressif\esp32/tools/sdk/esp32/include/esp_littlefs/include/esp_littlefs.h:16,
from --\hardware\espressif\esp32\libraries\LITTLEFS\src\LITTLEFS.cpp:23:
--\hardware\espressif\esp32/tools/sdk/esp32/include/newlib/platform_include/sys/termios.h:118: warning: "B1000000" redefined
#define B1000000 23

In file included from --\hardware\espressif\esp32\cores\esp32/Arduino.h:40,
from --\hardware\espressif\esp32\libraries\FS\src/FS.h:25,
from --\hardware\espressif\esp32\libraries\FS\src/vfs_api.h:18,
from --\hardware\espressif\esp32\libraries\LITTLEFS\src\LITTLEFS.cpp:17:
--\hardware\espressif\esp32\cores\esp32/binary.h:277: note: this is the location of the previous definition
#define B1000000 64

Also you need to apply:
882b12c
Otherwise dirctories cannot be removed Arduino-wise.

I will cleanup the LITTLEFS examples and try to do a PR at esp32s2 branch (with some hacky tools addition proposal to work with this https://github.com/lorol/arduino-esp32fs-plugin - of course you may need to do it in better way)

Observed huge RAM heap difference of my sketch between latest esp32s2 and master branches - about 70K more taken on s2. Board is TTGO T1

BrianPugh · 2020-11-03T15:46:55Z

@lorol I don't believe we need termioh.h, including it must have just been a mistake.

me-no-dev · 2020-11-03T17:50:09Z

@lorol I am aware of the ~70k issue. Need to figure out where those bytes went.

I also agree with @BrianPugh

me-no-dev · 2020-11-03T17:59:53Z

@lorol rmdir fix is applied. pull the changes

me-no-dev · 2021-04-16T23:31:04Z

closing now :)

Adding LITTLEFS w/o rebuild the IDF. Compatible with 3.3.

7a66810

based on https://github.com/joltwallet/esp_littlefs This will be my first PR, hope will do it right.

lorol mentioned this pull request Jun 17, 2020

Embed littleFS #3765

Closed

lorol added 3 commits June 18, 2020 07:15

cleanup and makeup package/package_esp32_index.template.json

99c854c

More tries to pass PR checkups ...

0602a04

Anyway, the LITTLEFS is a hack.

More tries to pass PR checkups ...

0ab506c

Anyway, the LITTLEFS is a hack.

lorol added a commit to lorol/arduino-esp32littlefs-plugin that referenced this pull request Jun 18, 2020

Added a note about PR to embed LITTLEFS to esp32 core is made, too se…

de505d9

…e the PR status here espressif/arduino-esp32#4096

lorol added a commit to lorol/LITTLEFS that referenced this pull request Jun 18, 2020

Added a note about PR to embed LITTLEFS to esp32 core is made, too se…

4d7eb3a

…e the PR status here: espressif/arduino-esp32#4096

atanisoft reviewed Jun 18, 2020

View reviewed changes

Corrections

4f1f7ad

atanisoft reviewed Jun 19, 2020

View reviewed changes

cleaner way

d056c40

BrianPugh mentioned this pull request Jun 19, 2020

Add LittleFS port as a component (IDFGH-3514) espressif/esp-idf#5469

Closed

lorol changed the title ~~Adding LITTLEFS w/o rebuild the IDF. Compatible with 3.3.~~ Adding LITTLEFS w/o rebuild the IDF. Compatible with IDF v3.3 Jun 19, 2020

lorol added 2 commits June 19, 2020 20:17

this value was changed on tests and not reverted, now back to default

72cf6b6

#define CONFIG_LITTLEFS_LOOKAHEAD_SIZE 128

- clean warnings sys/termios.h vs. binary.h (just removed the include)

5a7ac01

- In esp_littlefs.c, added a way to build for older IDF too (IDF 3.2 is used on release 1.0.4) #define CONFIG_LITTLEFS_FOR_IDF_3_2

BrianPugh mentioned this pull request Jun 24, 2020

Failed to get mtime attribute (-61) joltwallet/esp_littlefs#9

Closed

lorol added 8 commits July 13, 2020 10:06

Edited README with chages in this fork

abe9ea3

Modified Update library to allow FatFS OTA Updated file: package_esp32_index.template.json Added a sub-folder with IDE plugins

Combined plugin

6bf0962

Remove LITTLEFS - to be re-added as submodule from URL

c5646b0

Updated esp32fs.jar and README for it

Added LITTLEFS as module

5bd5698

LITTLEFS submodule updated

158f63d

LITTLEFS Sumbmodule from latest origin

82c6257

Merge branch 'master' of https://github.com/espressif/arduino-esp32.git

e9cee69

updated LITTLEFS to compile w/o warnings on esp32s2

056a5ee

lorol changed the title ~~Adding LITTLEFS w/o rebuild the IDF. Compatible with IDF v3.3~~ Adding LITTLEFS w/o rebuild the IDF Jul 30, 2020

BlueAndi mentioned this pull request Jul 30, 2020

Replace SPIFFS by LittleFS BlueAndi/Pixelix#8

Closed

lorol mentioned this pull request Sep 2, 2020

[Library Manager] Please add LittleFS_esp32 to the library manager arduino/Arduino#10719

Closed

Aircoookie mentioned this pull request Sep 12, 2020

SPIFFs is deprecated Aircoookie/WLED#1170

Closed

lorol mentioned this pull request Oct 15, 2020

Finally lorol/LITTLEFS#5

Closed

ericBcreator mentioned this pull request Oct 16, 2020

Question: is there a maximum number of controls? s00500/ESPUI#92

Closed

lorol mentioned this pull request Nov 3, 2020

Adding LITTLEFS after esp_littlefs (IDF) is built-in #4483

Merged

me-no-dev added the Type: For reference Common questions & problems label Nov 6, 2020

lorol mentioned this pull request Jan 15, 2021

Platformio and littlefs lorol/LITTLEFS#16

Closed

me-no-dev closed this Apr 16, 2021

		@@ -0,0 +1,130 @@
		// Copyright 2015-2016 Espressif Systems (Shanghai) PTE LTD

Adding LITTLEFS w/o rebuild the IDF #4096

Adding LITTLEFS w/o rebuild the IDF #4096

Conversation

lorol commented Jun 17, 2020

atanisoft left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorol commented Jun 18, 2020

BrianPugh commented Jun 18, 2020 • edited Loading

atanisoft commented Jun 18, 2020

BrianPugh commented Jun 18, 2020

atanisoft commented Jun 18, 2020

lorol commented Jun 18, 2020

atanisoft commented Jun 18, 2020

BrianPugh commented Jun 18, 2020

lorol commented Jun 18, 2020

BrianPugh commented Jun 18, 2020

lbernstone commented Jun 18, 2020

BrianPugh commented Jun 18, 2020

lbernstone commented Jun 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorol commented Jun 19, 2020 • edited Loading

lorol commented Sep 2, 2020

me-no-dev commented Nov 2, 2020

me-no-dev commented Nov 2, 2020

lorol commented Nov 2, 2020 • edited Loading

lorol commented Nov 3, 2020 • edited Loading

me-no-dev commented Nov 3, 2020

me-no-dev commented Nov 3, 2020

lorol commented Nov 3, 2020 • edited Loading

BrianPugh commented Nov 3, 2020

me-no-dev commented Nov 3, 2020

me-no-dev commented Nov 3, 2020

me-no-dev commented Apr 16, 2021

atanisoft left a comment •

edited

Loading

BrianPugh commented Jun 18, 2020 •

edited

Loading

lorol commented Jun 19, 2020 •

edited

Loading

lorol commented Nov 2, 2020 •

edited

Loading

lorol commented Nov 3, 2020 •

edited

Loading

lorol commented Nov 3, 2020 •

edited

Loading